Automated Machine Learning Frameworks for Radiomics: Comparative Evaluation Study

Introduction

In recent years, automated machine learning (AutoML) has emerged as a powerful approach to reducing technical barriers in machine learning model development. AutoML refers to the automation of the end-to-end machine learning workflow, including data preprocessing, feature selection, model selection, and hyperparameter optimization [1]. By delegating these technically demanding steps to algorithm-driven processes, AutoML allows clinicians and researchers without extensive machine learning experience to use advanced modeling techniques effectively, enabling medical professionals to concentrate on clinical questions and data interpretation instead of technical model building [2]. Modern AutoML frameworks promise improved model performance while promoting consistency and reproducibility by following structured, algorithm-driven processes that reduce individual variations. Similar to the introduction of genomics into clinical practice, medical professionals need transparent and interpretable models to understand radiomics features and apply them across the full spectrum of clinical end points, including classification tasks (eg, tumor grading and diagnosis), regression problems (eg, predicting continuous biomarkers or treatment response), and survival or time-to-event analyses that are essential in oncology research.

Radiomics represents a particularly challenging use case for AutoML in medical imaging. It involves the extraction of a large number of quantitative features from imaging exams for tissue characterization to serve as imaging biomarkers correlated with disease diagnosis or clinical end points [3]. The classical radiomics pipeline involves multiple sequential steps requiring domain-specific decisions and multidisciplinary expertise [4] (Figure 1) but faces significant limitations including a lack of reproducibility and standardization, where small variations in image acquisition or processing substantially affect extracted features [5,6]. Critical challenges include the need for harmonization methods to ensure feature consistency across different imaging platforms and institutions [7], and the high dimensionality of radiomics data relative to sample sizes, which creates highly correlated, sparse feature sets prone to overfitting that reduces feature robustness and biological interpretability across studies [8]. Numerous studies have shown the potential of radiomics features and radiomics-based models for cancer detection, tumor grading, and even predicting survival or treatment response, ultimately enhancing clinical decision-making [9-11]. Despite its potential, scaling radiomics into routine clinical practice remains challenging, and to date, no diagnostic tests or companion diagnostics based on radiomics have been successfully implemented.

‎

To address these challenges, several radiomics-specific AutoML frameworks have been developed to explicitly address the complexity of radiomics workflows, automating steps from feature extraction to model building [12,13]. In parallel, general-purpose AutoML frameworks, designed primarily for automating machine learning tasks on tabular data, have also been increasingly applied to radiomics studies. However, these efforts have largely remained method-specific, as most published studies have evaluated a single approach or a limited subset of tools in isolation [12-14]. Likewise, applications of general AutoML libraries to radiomics are often bespoke and tailored to specific use cases, without a unified methodological perspective across paradigms. [11]. As a result, current evidence provides only a fragmented view of how AutoML strategies behave in radiomics, particularly with respect to robustness, performance stability, and computational trade-offs across heterogeneous scenarios.

In this work, we explore the behavior of different AutoML frameworks in radiomics-based medical image classification. Using a unified experimental setup across 10 heterogeneous radiomics datasets, we compare the performance and computational characteristics of both general-purpose and radiomics-oriented AutoML frameworks across diverse imaging modalities, anatomical regions, and clinical end points. Our ultimate goal is to provide methodological insights that help guide the design and practical deployment of AutoML solutions in radiomics, supporting more robust, reproducible, and clinically relevant modeling workflows.

Methods

Ethical Considerations

This retrospective computational evaluation was conducted in accordance with the ethical principles of the Declaration of Helsinki. For the 8 publicly available datasets used, original study approvals apply [15,16]. For the private “Prostate” dataset, data were obtained from the ProCAncer-I project (grant 952159) [17]. Access was granted under the ProCAncer-I Data Sharing Agreement, which ensured full anonymization of the data and waived the requirement for informed consent for retrospective data use. For the private “Lung” dataset, given the use of retrospective and deidentified data, the study did not require institutional review board approval.

Datasets

This study used 10 distinct radiomics datasets to evaluate the behavior of AutoML frameworks in radiomics-based classification tasks. These datasets included diverse imaging modalities, anatomical regions, varying class balances, and clinical end points, thereby offering comprehensive coverage of the radiomics applications (Table 1 and Table S1 in Multimedia Appendix 1 [18,19]). Eight publicly available datasets from Open Radiomics (Brain Tumor Segmentation [BraTS] and The Cancer Imaging Archive [TCIA]) [15] and Workflow for Optimal Radiomics Classification (WORC; colorectal liver metastases [CRLM], Desmoid, gastrointestinal stromal tumors [GIST], Lipo, Liver, and Melanoma) databases were included [16]. Additionally, 2 private institutional datasets were added to test the generalization in radiomics contexts not represented in the public domain (Prostate [17] and Lung). Patient cohorts ranged significantly in size from 74 to 577 participants, reflecting variability between smaller single-institution datasets and larger multicenter collections. The inclusion and exclusion criteria for each cohort are detailed in their respective original publications [15-17]. To ensure standardized reporting and methodological transparency for the AI components of this study, we have provided the CLAIM (Checklist for Artificial Intelligence in Medical Imaging) reporting checklist [18] as Table S2 in the Multimedia Appendix 1.

All datasets provided preextracted radiomics features compliant with the Image Biomarker Standardization Initiative. Therefore, the present evaluation focuses exclusively on the modeling and classification stages of the radiomics workflow, and not on the full end-to-end pipeline including image preprocessing, segmentation, and feature extraction.

AutoML Frameworks

AutoML frameworks were selected based on two primary criteria: (1) open-source availability with no licensing fees and (2) prominence in current literature, defined here as recurrent use or discussion in recent peer-reviewed studies within the fields of medical imaging, radiomics, and AutoML. This selection aimed to represent both major AutoML paradigms, namely general-purpose frameworks originally developed for tabular data and domain-specific approaches tailored to radiomics. General-purpose AutoML methods included Autogluon (AWS AI, Amazon Web Services) [20], H2O AutoML (H2O.ai) [21], LightAutoML (Sber AI Lab) [22], MLjar (MLJAR) [23], PyCaret [24], and TPOT (EpistasisLab) [25], while radiomics-specific tools included AutoRadiomics [13], AutoML for Radiomics [26], AutoPrognosis (The van der Schaar Lab) [27], Simplatab [14], and WORC [12].

Evaluation Criteria

Frameworks Assessment

First, 3 qualitative aspects were assessed: obsolescence, accessibility, and explainability. Obsolescence was examined through repository activity and update frequency. Repository status was categorized as “active” (contributions within the last 6 months), “maintenance” (activity between 6 months and 2 years), or “obsolete” (inactive for more than 2 years), while update frequency was specifically rated as “high” (monthly releases), “moderate” (at least every 6 months), or “low” (sporadic updates less than once a year). Accessibility determined the barrier to entry based on installation complexity and required user expertise. Deployment quality was rated “high” for standard Python (Python Software Foundation) packages (accessible via pip, uv, or Docker) with comprehensive documentation, scaling down to “low” for complex builds with sparse guidance. Concurrently, the required learning curve was categorized from “low” for libraries adhering to standard conventions like the scikit-learn application programming interface, to “advanced,” which required deep expertise in the framework. Finally, interpretability was evaluated for the frameworks’ ability to provide model-level explanations of predictions, distinguishing between “advanced” integration of model-agnostic methods (eg, Shapley additive explanations and local interpretable model-agnostic explanations), “basic feature importance reporting,” and “none.” These evaluations were conducted by a data scientist with 4 years of experience developing radiomics-based predictive and prognostic algorithms. Frameworks deemed obsolete or exhibiting low accessibility were excluded from subsequent quantitative analysis.

Then, frameworks were quantitatively assessed based on their predictive performance using the area under the receiver operating characteristic curve (AUC) and computational efficiency based on the execution times during experiments. These 2 metrics were jointly analyzed to characterize the trade-off between predictive performance and computational efficiency across frameworks. An efficiency baseline was subsequently defined based on the joint assessment of predictive performance and computational runtime. Autogluon and MLjar offered different predefined performance configurations (presets) that were also evaluated as independent frameworks. Figure 2 summarizes the complete evaluation workflow.

‎

To ensure methodological consistency, all datasets underwent minimal preprocessing to allow each AutoML method to apply its specific training pipeline. Exceptions to this procedure occurred only when certain frameworks had inherent limitations: median imputation for Simplatab (missing value incompatibility) and numerical encoding for the Tree-Based Pipeline Optimization Tool (TPOT; categorical variable incompatibility). These exceptions were implemented to avoid excluding patients or variables, which would have altered the comparison across frameworks. All methods were initialized with predefined parameters and evaluated under the same hardware conditions with identical partitions. Performance was assessed using 5-fold nested cross-validation, reported as mean test AUC (SD) across the 5 folds. In each outer fold, the training partition was used for internal model development and optimization, whereas the held-out test partition was used for final performance estimation.

Additional details regarding radiomics feature extraction, framework pipelines, experimental setup and complementary metrics are provided in Table S3a and Table S3b in the Multimedia Appendix 1. All the code is available in GitHub [28].

Dataset Suitability for AutoML Evaluation in Radiomics

An effective dataset for comparative methodological evaluation should enable the measurement of consistent and discriminative signals of methodological differences. We adapted criteria from medical image segmentation validation to assess dataset suitability for AutoML in radiomics [29]. Two main requirements were defined to categorize a dataset as suitable for benchmarking: (1) low SD of AUC scores from the same method across the 5 folds (intramethod SD), which indicates statistical stability by ensuring that a method’s observed performance is consistent and not due to random chance or specific data splits; and (2) a high SD across different methods (intermethod SD), which indicates the presence of meaningful signals of methodological differences. Low intermethod SD would suggest that the task is too simple or that performance saturates quickly, limiting its utility for distinguishing methodological superiority.

The final suitability score was defined as the ratio between intermethod and intramethod SD. Conceptually rooted in the principles of ANOVA, this metric functions as a dimensionless signal-to-noise indicator of how clearly methodological differences can be detected. In this context, the intermethod SD acts as the “signal,” capturing the variance between the methods; while the intramethod SD represents the statistical “noise,” that is, the variance introduced by data resampling during the nested cross-validation. A ratio exceeding a parity threshold of 1.0 indicates that algorithmic superiority is distinct and distinguishable above the inherent noise. Datasets with low intramethod SD and high intermethod SD are preferred as they offer high differentiation power and result stability. Importantly, this metric should be interpreted as an exploratory heuristic rather than a formally validated measure.

Statistical Analysis

Statistical evaluation of performance differences was conducted using Python 3.11 (Python Software Foundation) and SciPy v.1.12 (SciPy community). Given the multiclassifier, multidataset design, we applied a nonparametric rank-based approach. The Friedman test assessed whether overall performance rankings differed significantly across the datasets, with global effect size quantified by Kendall W. Where significant differences were found (P<.05), pairwise comparisons were conducted using the Nemenyi post hoc test.

Overall mean AUC was reported as an unweighted macroaverage across all datasets to assess generalization and methodological robustness across clinically heterogeneous tasks, and median paired AUC differences and their 95% CIs were calculated to quantify performance gaps between frameworks.

Results

Dataset Suitability for Comparative Methodological Evaluation

Figure 3 summarizes the overall suitability of the evaluated datasets for comparative methodological analysis, using an exploratory metric defined as the ratio of intermethod to intramethod AUC SD.

As observed, datasets such as Lipo illustrated scenarios with a favorable balance between result stability and intermethod differentiation. Conversely, datasets like CRLM and Melanoma exhibited high intramethod variability, limiting the reliability of performance comparisons. In between these 2 extremes, Prostate exhibited intermediate behavior, with relatively higher internal variability than the most stable datasets. Finally, TCIA, GIST, Desmoid, Lung, Liver, and BraTS demonstrated stable results (low intramethod SD) but limited discriminatory capacity (low intermethod SD) between methods.

‎

Qualitative Assessment

Table 2 provides a summary of the qualitative characteristics for all initially considered frameworks. The assessment revealed limited viability for several radiomics-specific tools. Therefore, AutoML for Radiomics and AutoRadiomics were classified as obsolete, with inactive repositories and no recent updates, rendering them unusable in the experimental setup. WORC, while actively maintained, proved incompatible with the preextracted radiomics features used in this study and demanded a high level of programming expertise, deviating from the low-code paradigm typical of AutoML. AutoPrognosis, although functional and offering advanced capabilities, exhibited very high computational demands, failing to complete the first cross-validation fold within 72 hours on typical high-dimensional radiomics datasets, which led to its exclusion from the subsequent quantitative analysis. Consequently, only Simplatab remained as a viable radiomics-specific framework for the quantitative comparison.

In contrast, general-purpose frameworks such as LightAutoML, MLjar, and Autogluon demonstrated active maintenance, straightforward installation procedures, and lower requirements for machine learning expertise.

Notably, several frameworks, including MLjar, Simplatab, and TPOT, offered advanced interpretability tools (eg, Shapley Additive Explanations values and model bias analysis), a key differentiator for clinical translation and trustworthiness, whereas others provided only limited or no interpretability functionality.

Quantitative Performance and Efficiency

The quantitative evaluation was performed on the general-purpose frameworks and Simplatab, following the qualitative filtering described previously. Table 3 presents the detailed AUC results (mean, SD) for each framework across the 10 datasets, alongside the average and the median paired differences in performance compared with the top performer. While AUC was prioritized as a threshold-independent metric, complementary threshold-dependent clinical metrics (sensitivity, specificity, F₁-score, and balanced accuracy) were also computed in Table S4 in Multimedia Appendix 1. Consistent with the AUC findings, the complementary metrics reflected similar performance patterns across frameworks, although absolute values varied depending on the decision thresholds applied by each tool. However, no statistically significant differences were observed. Additionally, the aggregated results excluding CRLM and Melanoma are provided in Table S5 in Multimedia Appendix 1 for comparison, as these datasets were initially excluded due to limited compatibility across several frameworks.

Overall, Simplatab achieved the highest average AUC (mean 78.46%, SD 12.22%) across the heterogeneous datasets. The statistical analysis of the rankings confirmed Simplatab as the top-performing framework. Simplatab significantly outperformed several frameworks, including H2O AutoML, PyCaret, TPOT, LightAutoML, and specific presets of MLjar (explain and compete) and Autogluon (medium). Conversely, frameworks such as Autogluon (good [P=.37], high [P=.60], and best [P=.56] presets) and MLjar (perform [P=.70] and optuna [P=.51]) showed no statistically significant difference in overall ranking compared with the top-performing framework, although they yielded lower absolute mean AUCs and generally required longer execution times.

Analysis of the median paired differences (Δ vs Ref) revealed the clinical and technical performance gaps between methods. For instance, the gap between Simplatab and LightAutoML, which showed promise in stable datasets (Table S4 in Multimedia Appendix 1), widened when evaluated across the full heterogeneous suite, showing a median deficit of 3.9% (95% CI 2.7%-8.6%). H2O AutoML and PyCaret exhibited the most severe underperformance, with median gaps of 6.7% and 7.8%, respectively.

In line with the exploratory suitability analysis (Figure 3), CRLM and Melanoma yielded poor performance across most frameworks (AUC≤60% with high SD), reflecting the intrinsic instability of these datasets. Nevertheless, Simplatab showed relatively greater robustness on these challenging scenarios, achieving modestly higher AUC values (around 64.5% and 65.6%, respectively) with lower variance, which suggests improved tolerance to noisy and unstable data. In contrast, framework performance was relatively homogeneous in low-differentiation datasets, confirming their limited utility for methodological discrimination. For instance, AUC differences across frameworks in the Lung and TCIA datasets were minimal, further corroborating the findings derived from Figure 3.

Figure 4 presents the relationship between average AUC and runtime for each AutoML framework. From this joint perspective, a clear efficiency frontier emerges that is dominated by Simplatab. It occupies the most optimal position, providing the highest predictive performance with a moderate runtime (1.1 h). Fast alternatives, such as Autogluon medium, LightAutoML, and H2O, achieved speed at the cost of significant predictive accuracy. Meanwhile, the frameworks that achieved statistical parity with Simplatab in the ranking analysis (eg, Autogluon Best and MLjar optuna) fall behind on the efficiency frontier, requiring from 1.5 to 34 hours to complete the training process without surpassing Simplatab’s performance.

‎

Discussion

The comparative analysis of general-purpose and radiomics-specific AutoML frameworks for radiomics-based classification tasks reveals significant heterogeneity in performance, efficiency, computational costs, accessibility, and technical maturity, providing practical insights into the current capabilities of AutoML for radiomics applications.

A primary finding was Simplatab’s strong performance, achieving the highest average AUC across datasets (mean 78.46%, SD 12.22%), although differences in predictive performance were not consistently statistically significant compared with several general-purpose frameworks. Furthermore, it showed computational efficiency (1.1 h runtime) and relative resilience on challenging datasets such as CRLM and Melanoma, which exhibited high intramethod SD and poor performance across most other frameworks. Beyond performance, a notable characteristic of Simplatab is its accessibility and suite of tools, aimed at facilitating model inspection. The framework provides a graphical user interface that requires no coding expertise, together with built-in modules for model interpretability and basic assessments of data imbalance and model behavior. While these tools do not provide causal interpretability, they may represent practical mechanisms to increase model transparency and support user trust during model exploration and validation that may facilitate the gradual integration of radiomics-based models into clinical research workflows.

In contrast, general-purpose AutoML frameworks showed varying strengths and weaknesses. Intensive presets such as Autogluon (best and high quality) and MLjar (optuna) achieved statistical parity with Simplatab in overall dataset rankings; however, this competitive predictive performance came at the cost of substantially longer runtimes, which may limit their practicality for fast iteration. Autogluon medium quality, while emerging as an exceptionally fast option, exhibited a significant median performance gap (4.6%) compared with the top performer across the fully heterogeneous datasets. This positions Autogluon medium as a suitable tool for fast prototyping and preliminary analyses, but potentially suboptimal for final model selection. Conversely, H2O, despite being very fast, yielded lower predictive performance under default configurations, suggesting that its standard pipelines may be less suited for the complexities often inherent in high-dimensional radiomics datasets.

Our qualitative assessment highlighted significant limitations among several radiomics-specific tools. Frameworks such as AutoRadiomics and AutoML for Radiomics appeared to be inactive or technologically obsolete, reflecting the challenge of maintaining specialized software in a rapidly evolving machine learning landscape. WORC, while actively maintained, relied on configuration files and scripting that demanded considerable programming expertise, which conflicts with the low-code or no-code paradigm typically associated with AutoML solutions. AutoPrognosis, despite its advanced methodological scope, proved computationally infeasible for the high-dimensional datasets considered in this study under the adopted experimental constraints. Consequently, Simplatab was the only radiomics-specific framework that could be included under the constraints of this study.

The dataset suitability analysis revealed limitations in several public radiomics datasets: CRLM and Melanoma showed high intramethod variability, making it difficult to discern true performance differences from statistical noise, whereas others (Lung, TCIA, and Liver) showed low intermethod variability, which limits their ability to discriminate between modeling pipelines. These findings suggest the need for larger, more diverse, and statistically stable radiomics datasets to support meaningful methodological comparisons and advancements.

Our study has limitations that highlight critical gaps in the current AutoML landscape for radiomics. First, the present evaluation focused exclusively on classification tasks. The absence of robust, user-friendly AutoML frameworks supporting survival analysis represents a major bottleneck for oncology, where time-to-event end points are paramount. Future development should prioritize the integration of survival models into accessible AutoML packages. In addition, because the primary objective of this study was methodological comparison across heterogeneous datasets rather than specific clinical deployment, our evaluation relied primarily on threshold-independent discrimination (AUC). Assessing true clinical translation requires calibration metrics and decision-curve analysis, which depend on disease-specific contexts and operational thresholds. The absence of these context-dependent metrics remains a limitation, and future studies must evaluate calibration and net clinical benefit. Second, the use of pre-extracted features isolated the modeling component of the radiomics pipeline but does not address the ultimate goal of achieving a fully automated, end-to-end workflow. As a consequence, the exclusion of some tools due to compatibility or usability constraints may introduce a degree of selection bias. Additionally, minor framework-specific preprocessing adaptations (eg, imputation or encoding) were strictly required to prevent the exclusion of patients. While these adaptations were necessary to ensure that all frameworks were evaluated on the same cohorts, we acknowledge that such preprocessing differences may have influenced performance to some extent; however, their impact was not formally evaluated and therefore represents a limitation for the comparative study. A major unmet need is the development of frameworks that also integrate upstream steps such as feature extraction, harmonization, and reproducibility control, which remain critical barriers for clinical translation and are not within the scope of this study. While tools like Simplatab demonstrate promise at the modeling stage, to ensure reliability and clinical trust, the field should progress toward holistic solutions that automate the entire radiomics workflow, from image to prediction. Finally, it is important to highlight that these findings should be interpreted within the context of an internal benchmarking study, as no external validation was performed.

In conclusion, AutoML frameworks offer substantial potential to accelerate and democratize radiomics research by automating complex modeling tasks. While no single framework demonstrated absolute predictive superiority, Simplatab emerges as a promising tool for users prioritizing ease of use, model inspection, and strong predictive performance with reasonable computational cost. General-purpose tools such as the Autogluon medium preset provide efficient options for rapid experimentation, while heavier presets from Autogluon or MLjar achieved statistically comparable performance to Simplatab, but they required substantially higher computational times, highlighting the importance of the efficiency trade-offs. However, substantial challenges remain. The field requires more stable and diverse radiomics datasets, AutoML solutions capable of supporting survival analysis and, critically, frameworks that enable robust harmonization and reproducibility across the full radiomics workflow are required. Although these tools represent a step forward for the modeling stage, considerable development is still required to achieve fully automated, reliable, and clinically translatable radiomics pipelines.